Image reconstruction method and device

ABSTRACT

Embodiments of this application provide an image reconstruction method and device. The method includes: inputting a first image into a newly constructed super-resolution model to obtain a reconstructed second image, where a resolution of the second image is higher than that of the first image; the newly constructed super-resolution model is obtained by training an initial super-resolution model by using an error loss; the error loss includes a pixel mean square error and an image feature mean square error; and an image feature includes at least one of a texture feature, a shape feature, a spatial relationship feature, and an image high-level semantic feature. According to the embodiments of this application, quality of a reconstructed image can be improved.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/CN2018/120447, filed on Dec. 12, 2018, which claims priority toChinese Patent Application No. 201711387428.6, filed on Dec. 20, 2017.The disclosures of the aforementioned applications are herebyincorporated by reference in their entireties.

TECHNICAL FIELD

Embodiments of the present invention relate to the field ofcommunications technologies, and in particular, to an imagereconstruction method and device.

BACKGROUND

Image super-resolution reconstruction refers to a technology thatreconstructs a high-resolution image from a low-resolution image byusing an image processing method, can effectively improve an imagedefinition, and has great significance in fields such as videosurveillance, camera photographing, high-definition televisions, andmedical images. In image super-resolution reconstruction, face imagesuper-resolution (face image super-resolution) reconstruction is widelyapplied, and the face image super-resolution reconstruction is alsoreferred to as face hallucination (face hallucination).

Currently, a face image super-resolution reconstruction method includesa signal reconstruction-based method and a machine learning-basedmethod. The signal reconstruction-based method is mainly implemented byusing a signal reconstruction theory in the signal processing field, forexample, Fourier transform and polynomial interpolation. The signalreconstruction-based method is usually easy to implement, but areconstructed image loses much detail information, and has blurred edgesand obvious jagged edges.

The machine learning-based method is to input a low-resolution image,and then reconstruct the low-resolution image by using asuper-resolution model, to obtain a reconstructed image obtained throughmaximum a posteriori probability estimation. The super-resolution modelused in the machine learning-based method is obtained by training aninitial super-resolution model. A training process of thesuper-resolution model is a process of adjusting a parameter in thesuper-resolution model based on a pixel mean square error between ahigh-resolution image and an image obtained by reconstructing thelow-resolution image. However, when image reconstruction is performed byusing the super-resolution model that is obtained through training basedon only the pixel mean square error, a generated image is evidentlysmoothed and loses much high-frequency information.

SUMMARY

Embodiments of this application disclose an image reconstruction methodand device, to improve image reconstruction quality.

According to a first aspect, an embodiment of this application providesan image reconstruction method. The method includes: inputting a firstimage into a newly constructed super-resolution model to obtain areconstructed second image, where a resolution of the second image ishigher than that of the first image; the newly constructedsuper-resolution model is obtained by training an initialsuper-resolution model by using an error loss; the error loss includes apixel mean square error and an image feature mean square error; and animage feature includes at least one of a texture feature, a shapefeature, a spatial relationship feature, and an image high-levelsemantic feature. In a training phase of the super-resolution model, theerror loss includes the pixel mean square error, and the error loss alsoincludes the image feature mean square error. The error loss used totrain the initial super-resolution model includes more comprehensiveerror loss information. Therefore, the newly constructedsuper-resolution model that is obtained through training and that isused for image reconstruction is more accurate, so that a loss ofhigh-frequency information of a reconstructed image can be reduced, andreconstruction quality of the reconstructed image can be improved.

In an embodiment, the error loss is an error loss between a third imageand a fourth image, and the third image is obtained throughreconstruction after inputting a fifth image into the initialsuper-resolution model; the fourth image is a high-resolution image, andthe fifth image is a low-resolution image obtained by performingblurring processing on the fourth image; and the initialsuper-resolution model is used to reconstruct an image input into theinitial super-resolution model, to improve a resolution.

In an embodiment, there are M third images, M fourth images, and M fifthimages, there are M error losses, and the M third images are obtainedthrough reconstruction after inputting the M fifth images into theinitial super-resolution model; the M error losses are determined basedon the M third images and the M fourth images; and any one of the Merror losses is an error loss between an i^(th) third image in the Mthird images and a j^(th) fourth image in the M fourth images, an imageobtained after a fifth image obtained by performing blurring processingon the j^(th) fourth image is input into the initial super-resolutionmodel is the i^(th) third image, M is a positive integer greater than 1,and i and j each are a positive integer less than or equal to M. When Mis a positive integer greater than 2, the initial super-resolution modelis adjusted by using a plurality of error losses obtained by using aplurality of groups of training samples, so that more sample informationis provided for adjusting the initial super-resolution model, and thenewly constructed super-resolution model obtained through adjustment hashigher precision. In addition, if there are a plurality of pairs oftraining samples, and each time an error loss is obtained, the initialsuper-resolution model is adjusted based on the error loss, an excessivequantity of adjustment times causes wastes of processing resources andstorage resources. However, adjusting the initial super-resolution modelby using the plurality of error losses obtained by using the pluralityof groups of training samples can reduce a quantity of times ofadjusting a parameter in the super-resolution model. Therefore,processing resources and storage resources can be saved.

In an embodiment, the newly constructed super-resolution model isobtained by adjusting a parameter in the initial super-resolution modelbased on the M error losses; or the initial super-resolution model isthe first super-resolution model, a parameter in the firstsuper-resolution model is adjusted based on the first error loss in theM error losses to obtain the second super-resolution model, a parameterin an r^(th) super-resolution model is adjusted based on an r^(th) errorloss to obtain an (r+1)^(th) super-resolution model, and the newlyconstructed super-resolution model is obtained by adjusting a parameterin an M^(th) super-resolution model by using an M^(th) error loss, wherer is a positive integer greater than or equal to 1 and less than orequal to M.

In an embodiment, the initial super-resolution model includes nsuper-resolution submodels, and n is a positive integer greater than orequal to 2; the super-resolution submodel is used to reconstruct imageinformation input into the super-resolution submodel, to improve aresolution; the image information includes pixel value information andimage feature information; in the n super-resolution submodels, an inputof the first super-resolution submodel is the first image, an output ofthe first super-resolution submodel is used as an input of the secondsuper-resolution submodel, an output of a (t−1)^(th) super-resolutionsubmodel is used as an input of a t^(th) super-resolution submodel, andan output of the t^(th) super-resolution submodel is used as an input ofa (t+1)^(th) super-resolution submodel; t is a positive integersatisfying 2≤t≤n−1; and the output of the t^(th) super-resolutionsubmodel is used as an input of an output synthesis module, an output ofthe output synthesis module is used as an input of an n^(th)super-resolution submodel, an output of the n^(th) super-resolutionsubmodel is the second image, and the output synthesis module isconfigured to determine the input of the n^(th) super-resolutionsubmodel based on reconstructed image information output by the firstn−1 super-resolution submodels and a weight of each piece of the outputreconstructed image information. When n is a positive integer greaterthan 1, a plurality of super-resolution submodels are cascaded toreconstruct a low-resolution image. A pixel value of a reconstructedimage obtained through reconstruction is higher, so that image qualityof the reconstructed image can be improved. In addition, reconstructedimages output by the first n−1 super-resolution submodels are all usedas image information input into the last super-resolution submodel, andinclude more image information, so that image information loss isreduced. This can improve image reconstruction quality by improvingprecision of the newly constructed super-resolution model.

In an embodiment, the reconstructed image information output by theoutput synthesis module O_(S)=Σ_(k=1) ^(n−1)(w_(k)O_(k)), and k is apositive integer satisfying 1≤k≤n−1; and w_(k) is a weight of a k^(th)super-resolution submodel.

In an embodiment, w_(k) is the parameter in the initial super-resolutionmodel. In a training process of the initial super-resolution model, aweight w_(k) of the initial super-resolution model may be optimizedbased on the error loss.

In an embodiment, the super-resolution submodel is a three-layer fullyconvolutional deep neural network. In the foregoing three-layer fullyconvolutional deep neural network, the first convolution layer and thesecond convolution layer are used to extract image information from alow-resolution image, that is, obtain information that can be used forsuper-resolution reconstruction. The third convolution layerreconstructs a high-resolution image by using the image informationextracted and transformed by the first two layers. The two additionalconvolution layers in the three-layer fully convolutional deep neuralnetwork can help extract more precise image information than extractingthe image information by using only one convolution layer. In addition,the super-resolution submodels constituted by three-layer fullyconvolutional deep neural networks need to be cascaded to constitute thesuper-resolution model, and cascading a plurality of super-resolutionsubmodels requires more calculation resources, but a relatively smallquantity of convolution layers indicates a relatively low calculationamount. Therefore, a tradeoff between calculation resources andprecision needs to be considered for a quantity of convolution layers inthe super-resolution submodels. When the super-resolution submodel usesthe three-layer fully convolutional deep neural network, more preciseimage information can be extracted by using fewer calculation resources.The more precise image information helps reconstruct a higher-qualityreconstructed image and save calculation resources.

In an embodiment, the error loss L=λ₁L1+λ₂L2+λ₃L3, where L1 is the pixelmean square error, λ₁ is a weight of the pixel mean square error, L2 isthe image feature mean square error, λ₂ is a weight of the image featuremean square error, L3 is a regularization term of w_(k), and λ₃ is aweight of the regularization term. The added regularization term L3 isused to reduce overfitting, improve precision of the newly constructedsuper-resolution model, and improve quality of the reconstructed image.

In an embodiment, each convolution layer in the three-layer fullyconvolutional deep neural network includes at least one convolutionkernel, and a weight matrix W of the convolution kernel is a parameterin the initial super-resolution model.

According to a second aspect, an embodiment of this application providesan image reconstruction device, including a processor and a memory. Thememory is configured to store a program instruction, and the processoris configured to invoke the program instruction to perform the followingoperations: inputting a first image into a newly constructedsuper-resolution model to obtain a reconstructed second image, where aresolution of the second image is higher than that of the first image;the newly constructed super-resolution model is obtained by training aninitial super-resolution model by using an error loss; the error lossincludes a pixel mean square error and an image feature mean squareerror; and an image feature includes at least one of a texture feature,a shape feature, a spatial relationship feature, and an image high-levelsemantic feature. In a training phase of the super-resolution model, theerror loss includes the pixel mean square error, and the error loss alsoincludes the image feature mean square error. The error loss used totrain the initial super-resolution model includes more comprehensiveerror loss information, so that a loss of high-frequency information ofa reconstructed image can be reduced, and reconstruction quality of thereconstructed image can be improved.

In an embodiment, the error loss is an error loss between a third imageand a fourth image, and the third image is obtained throughreconstruction after inputting a fifth image into the initialsuper-resolution model; the fourth image is a high-resolution image, andthe fifth image is a low-resolution image obtained by performingblurring processing on the fourth image; and the initialsuper-resolution model is used to reconstruct an image input into theinitial super-resolution model, to improve a resolution.

In an embodiment, there are M third images, M fourth images, and M fifthimages, there are M error losses, and the M third images are obtainedthrough reconstruction after inputting the M fifth images into theinitial super-resolution model; the M error losses are determined basedon the M third images and the M fourth images; and any one of the Merror losses is an error loss between an i^(th) third image in the Mthird images and a j^(th) fourth image in the M fourth images, an imageobtained after a fifth image obtained by performing blurring processingon the j^(th) fourth image is input into the initial super-resolutionmodel is the i^(th) third image, M is a positive integer greater than 1,and i and j each are a positive integer less than or equal to M. When Mis a positive integer greater than 2, the initial super-resolution modelis adjusted by using a plurality of error losses obtained by using aplurality of groups of training samples, so that more sample informationis provided for adjusting the initial super-resolution model, and thenewly constructed super-resolution model obtained through adjustment hashigher precision.

In an embodiment, the newly constructed super-resolution model isobtained by adjusting a parameter in the initial super-resolution modelbased on the M error losses; or

the initial super-resolution model is the first super-resolution model,a parameter in the first super-resolution model is adjusted based on thefirst error loss in the M error losses to obtain the secondsuper-resolution model, a parameter in an r^(th) super-resolution modelis adjusted based on an r^(th) error loss to obtain an (r+1)^(th)super-resolution model, and the newly constructed super-resolution modelis obtained by adjusting a parameter in an M^(th) super-resolution modelby using an M^(th) error loss, where r is a positive integer greaterthan or equal to 1 and less than or equal to M.

In an embodiment, the initial super-resolution model includes nsuper-resolution submodels, and n is a positive integer greater than orequal to 2; the super-resolution submodel is used to reconstruct imageinformation input into the super-resolution submodel, to improve aresolution; the image information includes pixel value information andimage feature information; in the n super-resolution submodels, an inputof the first super-resolution submodel is the first image, an output ofthe first super-resolution submodel is used as an input of the secondsuper-resolution submodel, an output of a (t−1)^(th) super-resolutionsubmodel is used as an input of a t^(th) super-resolution submodel, andan output of the t^(th) super-resolution submodel is used as an input ofa (t+1)^(th) super-resolution submodel; t is a positive integersatisfying 2≤t≤n−1;

and the output of the t^(th) super-resolution submodel is used as aninput of an output synthesis module, an output of the output synthesismodule is used as an input of an n^(th) super-resolution submodel, anoutput of the n^(th) super-resolution submodel is the second image, andthe output synthesis module is configured to determine the input of then^(th) super-resolution submodel based on reconstructed imageinformation output by the first n−1 super-resolution submodels and aweight of each piece of the output reconstructed image information. Whenn is a positive integer greater than 1, a plurality of super-resolutionsubmodels are cascaded to reconstruct a low-resolution image. A pixelvalue of a reconstructed image obtained through reconstruction ishigher, so that image quality of the reconstructed image can beimproved. In addition, reconstructed images output by the first n−1super-resolution submodels are all used as image information input intothe last super-resolution submodel, and include more image information,so that image information loss is reduced. This can improve precision ofthe newly constructed super-resolution model, and improve imagereconstruction quality.

In an embodiment, the reconstructed image information output by theoutput synthesis module

${O_{S} = {\sum\limits_{k = 1}^{n - 1}\left( {w_{k}O_{k}} \right)}},$

and k is a positive integer satisfying 1≤k≤n−1; and w_(k) is a weight ofa k^(th) super-resolution submodel.

In an embodiment, w_(k) is the parameter in the initial super-resolutionmodel. In a training process of the initial super-resolution model, aweight w_(k) of the initial super-resolution model may be optimizedbased on the error loss.

In an embodiment, the super-resolution submodel is a three-layer fullyconvolutional deep neural network. In the foregoing three-layer fullyconvolutional deep neural network, the first layer and the second layerare used to extract image information from a low-resolution image, thatis, obtain information that can be used for super-resolutionreconstruction. The third layer reconstructs a high-resolution image byusing the image information extracted and transformed by the first twolayers. The two additional convolution layers in the three-layer fullyconvolutional deep neural network can help extract more precise imageinformation than extracting the image information by using only oneconvolution layer. In addition, the super-resolution submodelsconstituted by three-layer fully convolutional deep neural networks needto be cascaded to constitute the super-resolution model, and cascading aplurality of super-resolution submodels requires more calculationresources, but a relatively small quantity of convolution layersindicates a relatively low calculation amount. Therefore, a tradeoffbetween calculation resources and precision needs to be considered for aquantity of convolution layers in the super-resolution submodels. Whenthe super-resolution submodel uses the three-layer fully convolutionaldeep neural network, more precise image information can be extracted byusing fewer calculation resources. The more precise image informationhelps reconstruct a higher-quality reconstructed image and savecalculation resources.

In an embodiment, the error loss L=λ₁L1+λ₂L2+λ₃L3, where L 1 is thepixel mean square error, λ₁ is a weight of the pixel mean square error,L2 is the image feature mean square error, λ₂ is a weight of the imagefeature mean square error, L3 is a regularization term of w_(k), and λ₃is a weight of the regularization term. The added regularization term L3is used to reduce overfitting, improve precision of the newlyconstructed super-resolution model, and improve quality of thereconstructed image.

In an embodiment, each convolution layer in the three-layer fullyconvolutional deep neural network includes at least one convolutionkernel, and a weight matrix W of the convolution kernel is a parameterin the initial super-resolution model.

According to a third aspect, an embodiment of this application providesan image reconstruction device. The device includes a module or a unitconfigured to perform the image reconstruction method provided in anyone of the first aspect or the possible implementations of the firstaspect.

According to a fourth aspect, an embodiment of the present inventionprovides a chip system. The chip system includes at least one processor,a memory, and an interface circuit. The memory, the interface circuit,and the at least one processor are interconnected by using a line, andthe at least one memory stores a program instruction. When the programinstruction is executed by the processor, the method described in anyone of the first aspect or the possible implementations of the firstaspect is implemented.

According to a fifth aspect, an embodiment of the present inventionprovides a computer-readable storage medium. The computer-readablestorage medium stores a program instruction, and when the programinstruction is run by a processor, the method described in any one ofthe first aspect or the possible implementations of the first aspect isimplemented.

According to a sixth aspect, an embodiment of the present inventionprovides a computer program product. When the computer program productis run by a processor, the method described in any one of the firstaspect or the possible implementations of the first aspect isimplemented.

In the training phase of the super-resolution model, the error lossincludes the pixel mean square error, and the error loss also includesthe image feature mean square error. The error loss used to train theinitial super-resolution model includes more comprehensive error lossinformation, so that a loss of high-frequency information of thereconstructed image can be reduced, and reconstruction quality of thereconstructed image can be improved. n super-resolution submodels areused. When n is a positive integer greater than 1, a plurality ofsuper-resolution submodels are cascaded to reconstruct thelow-resolution image. The pixel value of the reconstructed imageobtained through reconstruction is higher, so that the image quality ofthe reconstructed image can be improved. In addition, the reconstructedimages output by the first n−1 super-resolution submodels are all usedas the image information input into the last super-resolution submodel,and include more image information, so that image information loss isreduced. This can improve the image reconstruction quality by improvingthe precision of the newly constructed super-resolution model.

BRIEF DESCRIPTION OF DRAWINGS

The following describes the accompanying drawings used in theembodiments of this application.

FIG. 1 is a schematic flowchart of an image reconstruction methodaccording to an embodiment of this application;

FIG. 2 is a schematic block diagram of a method for constructing animage reconstruction model according to an embodiment of thisapplication;

FIG. 3 is a schematic structural diagram of a super-resolution modelaccording to an embodiment of this application;

FIG. 4 is a schematic structural diagram of a super-resolution submodelaccording to an embodiment of this application;

FIG. 5 is a schematic structural diagram of an image reconstructiondevice according to an embodiment of this application; and

FIG. 6 is a schematic structural diagram of another image reconstructiondevice according to an embodiment of this application.

DESCRIPTION OF EMBODIMENTS

First, to facilitate understanding of the embodiments of thisapplication, some concepts or terms in the embodiments of thisapplication are explained.

(1) Super-Resolution

Super-resolution (SR) refers to a technology that reconstructs ahigh-resolution (high resolution, HR) image from a low-resolution (lowresolution, LR) image by using an image processing method and by using acomputer. The high-resolution image means that the image has a highpixel density, and can provide more image details. These details usuallyplay a key role during application.

Image super-resolution technologies can be classified into two types: areconstruction-based image super-resolution method and a learning-basedimage super-resolution method. In the reconstruction-based imagesuper-resolution method, a high-resolution image with a maximum aposteriori probability may be obtained through statistics collection andestimation by using a frequency domain algorithm or a space domainalgorithm. The learning-based image super-resolution method may includetwo phases: a training phase and a test phase.

In the training phase, an initial super-resolution model and a trainingset are first established. The training set may include a plurality oflow-resolution images and a high-resolution image corresponding to eachlow-resolution image. The low-resolution images and the correspondinghigh-resolution images in the training set are used to learn acorrespondence between the high-resolution images and the low-resolutionimages, to correct a value of a parameter in the initialsuper-resolution model, so as to converge an error between thehigh-resolution image and a reconstructed image. Finally, a newlyconstructed super-resolution model obtained after training isdetermined. In the test phase, super-resolution reconstruction of animage may be guided by using the newly constructed super-resolutionmodel.

A method for obtaining the low-resolution image and the high-resolutionimage corresponding to the low-resolution image may be: processing thehigh-resolution image by using a fuzzy function, to obtain thecorresponding low-resolution image.

The initial super-resolution model may be a model determined based on anexperiment, and may be non-linear. The super-resolution model may be aconvolutional neural network.

(2) Convolutional Neural Network

The neural network may include a neural unit. The neural unit may be anoperation unit that uses x_(s) and an intercept 1 as an input, and anoutput of the operation unit may be as follows:

h _(W,b)(x)=f(W ^(T) x)=f(Σ_(s=1) ^(n) W _(s) x _(s) +b)   (1-1)

s=1, 2, . . . , or n, n is a natural number greater than 1, Ws is aweight of x_(s), and b is an offset of the neural unit. f is anactivation function (activation functions) of the neural unit, and isused to introduce a non-linear feature to the neural network, to convertan input signal in the neural unit into an output signal. The outputsignal of the activation function may be used as an input of a nextconvolutional layer. The activation function may be a sigmoid function.The neural network is a network constituted by joining many singleneural units together, to be specific, an output of a neural unit may bean input of another neural unit. An input of each neural unit may beconnected to a local receptive field of a previous layer to extract afeature of the local receptive field. The local receptive field may be aregion including several neural units.

The convolutional neural network (CNN) is a deep neural network with aconvolutional structure. The convolutional neural network includes afeature extractor including a convolutional layer and a sub-samplinglayer. The feature extractor may be considered as a filter. Aconvolution process may be considered as performing convolution on aninput image or a convolution feature map (feature map) by using atrainable filter. The convolutional layer is a neuron layer that is inthe convolutional neural network and that performs convolutionprocessing on an input signal. At the convolutional layer in theconvolutional neural network, a neuron may be connected only to someadjacent-layer neurons. One convolutional layer usually includes severalfeature maps, and each feature map may include some neural unitsarranged in a rectangle shape. The neural units on a same feature mapshare a weight. The shared weight herein is a convolution kernel. Weightsharing may be understood as that a manner of extracting imageinformation is irrelevant to a location. The principle implied herein isthat statistical information of a part of an image is the same as thatof another part. To be specific, image information that is learned in apart can also be used in another part. Therefore, image informationobtained through same learning can be used for all locations on theimage. In a same convolutional layer, a plurality of convolution kernelsmay be used to extract different image information. Usually, a largerquantity of convolution kernels indicates richer image informationreflected by a convolution operation.

A convolution kernel may be initialized in a form of a random-sizematrix. A proper weight may be obtained by a convolution kernel throughlearning in a training process of the convolutional neural network. Inaddition, a direct benefit brought by weight sharing is to reduce aconnection between layers of the convolutional neural network, andfurther reduce an overfitting risk.

(3) Back Propagation Algorithm

The convolutional neural network may correct the value of the parameterin the initial super-resolution model in the training process by usingan error back propagation (BP) algorithm, so that a reconstruction errorloss of the super-resolution model becomes small. Specifically, an errorloss is caused when a signal is input and output by using forwardpropagation, and the parameter in the initial super-resolution model isupdated by using back propagation error loss information, so that theerror loss is converged. The back propagation algorithm is anerror-loss-centered back propagation motion intended to obtain aparameter, such as a weight matrix, of an optimal super-resolutionmodel.

(4) Pixel Value and Image Feature

Pixel value information and image feature information of an image may becollectively referred to as image information.

The pixel value may be a red green blue (RGB) color value, and the pixelvalue may be a long integer indicating a color. For example, the pixelvalue is 65536*Red+256*Green+Blue, where Blue represents a bluecomponent, Green represents a green component, and Red represents a redcomponent. In the color components, a smaller value indicates lowerbrightness, and a larger value indicates higher brightness. For agrayscale image, the pixel value may be a grayscale value.

The image feature includes a texture feature, a shape feature, a spatialrelationship feature, and an image high-level semantic feature. Detailsare described as follows:

The texture feature of the image is a global feature of the image, anddescribes a surface property of a scene corresponding to the image or animage region. The texture feature of the image is not a feature that isbased on a single pixel, but is a feature obtained through statisticscollection and calculation in a region including a plurality of pixels.As a statistical feature, the texture feature of the image has a strongresistance to noises. However, when a resolution of the image changes,the texture feature of the image may have a relatively large deviation.The texture feature of the image may be described by using the followingmethods: a. A statistical method is, for example, extracting a texturefeature from an autocorrelation function of the image (or a powerspectral function of the image), and extracting feature parameters suchas a thickness and directivity of a texture by calculating the powerspectral function of the image. b. A geometric method is a texturefeature analysis method that is based on a theory of a texture primitive(a basic texture element). In this method, a complex texture feature maybe constituted by several simple texture primitives that are repeatedlyarranged in a regular form. c. A model method is based on an imageconstruction model, and uses a parameter of the model to represent thetexture feature.

The shape feature of the image may have two types of representationmethods: a contour feature and a region feature. The contour feature ofthe image is a contour of an outer boundary of an object, and the regionfeature of the image is an entire shape region occupied by the object.The shape feature of the image may be described by using the followingmethods: a. A boundary feature method is to obtain a shape parameter ofthe image by describing a boundary feature. b. The Fourier shapedescriptor method is to express the shape feature by using a Fouriertransform of the object boundary as a shape description, and using aclosure property and periodicity of the region boundary to derive acurvature function, a centroid distance, and a complex coordinatefunction from boundary points. c. A geometric parameter method is to usea region feature description method for shape expression and matching.For example, a shape parameter matrix, an area, a boundary length, andthe like are used to describe the shape feature of the image.

The spatial relationship feature of the image is a mutual spatiallocation relationship or a relative direction relationship among aplurality of regions obtained by segmenting the image. Theserelationships may also be classified into a connection relationship, anadjacency relationship, an overlapping relationship, an overlappingrelationship, an inclusion relationship, an inclusion relationship, andthe like. Usually, spatial locations of an image may be classified intotwo types: a relative spatial location and an absolute spatial location.The relative spatial location emphasizes a relative location amongtargets, for example, an up-down and left-right relationship, and thelike. The absolute spatial location emphasizes a distance andorientation among the targets. Use of the spatial relationship featureof the image can enhance an ability of describing and distinguishingimage content, but the spatial relationship feature is usually sensitiveto rotation, inversion and scale change of the image or the object.

Compared with the texture feature, the shape feature, and the spatialrelationship feature of the image, the image high-level semantic featureis a higher-level cognitive feature used to describe human understandingof the image. The image high-level semantic feature is to determine, byusing the image as an object, a target at a location in the image, amutual relationship among target scenarios, a scenario of the image, andhow to apply the scenario. Extracting the image high-level semanticfeature is a process of converting an input image into a text-likelanguage expression that can be intuitively understood. A correspondencebetween an image and a semantic text needs to be established to obtainthe image high-level semantic feature.

Based on an abstraction degree of a combination of semantic elements inthe image, image high-level semantic features may be classified into anobject semantic feature, a spatial relationship semantic feature, ascenario semantic feature, a behavior semantic feature, an emotionsemantic feature, and the like. The object semantic feature may be afeature used to determine a person, an animal, a physical object, andthe like. The spatial relationship semantic feature may be, for example,a semantic feature used to determine that “a person is in front of ahouse” or “a ball is on a grass”. The scenario semantic feature may be,for example, a semantic feature used to determine “sea” or “awilderness”. The behavior semantic feature may be, for example, asemantic feature used to determine “a dance performance” or “a sportscompetition”. The emotion semantic feature may be, for example, asemantic feature used to determine “a pleasant image” or “an excitingimage”. The object semantic feature and the spatial relationshipsemantic feature need to be logically inferred and a type of the targetin the image needs to be identified. The scenario semantic feature, thebehavior semantic feature, and the emotion semantic feature relate to anabstract property of the image, and high-level inference needs to beperformed on a meaning of the feature of the image.

It may be understood that the foregoing examples of the image high-levelsemantic feature are merely used to explain the image high-levelsemantic feature in the embodiments of this application, and should notconstitute a limitation.

Based on different sources of the image high-level semantic feature, amethod for extracting the image high-level semantic feature may includea processing range-based method, a machine learning-based method, ahuman-computer interaction-based method, and an external informationsource-based method. The processing range-based method may be performedon a premise of image segmentation and object identification. Semanticsmining is performed by using an object template, a scenario classifier,and the like and by identifying objects and a topology relationshipamong the objects, to generate corresponding scenario semanticinformation. The machine learning-based method is to learn a low-levelfeature of the image, and mine an association between the low-levelfeature and image semantics, to establish a mapping relationship betweenthe low-level feature and the high-level semantic feature of the image.The machine learning-based method mainly includes two key steps: Thefirst is extraction of the low-level feature, such as a texture and ashape. The second is the application of a mapping algorithm. In thehuman-computer interaction-based method, a system usually uses thelow-level feature, and a user adds high-level knowledge. The extractionmethod mainly includes two aspects: image preprocessing and feedbacklearning. An image preprocessing manner may be manually labeling imagesin an image library, or may be some automatic or semi-automatic imagesemantic labeling methods. Feedback learning is to add manualintervention to the process of extracting image semantics, extractsemantic features of the image through repeated interactions between theuser and the system, and establish and correct high-level semanticconcepts associated with image content.

(5) Error Loss

An error loss caused when a signal is input and output by using forwardpropagation may include a pixel mean square error and a feature loss.The feature loss may be an image feature mean square error, and an imagefeature may include at least one of a texture feature, a shape feature,a spatial relationship feature, and an image high-level semanticfeature. The following separately describes the pixel mean square errorand the image feature mean square error.

A reconstructed image of an input low-resolution image may be obtainedbased on an initial model, and a pixel mean square error, namely, apixel mean square error loss, between the reconstructed image and ahigh-resolution image corresponding to the input low-resolution imagemay be calculated:

$\begin{matrix}{{L1} = {\frac{1}{FH}{\sum\limits_{x = 1}^{F}{\sum\limits_{y = 1}^{H}\left( {I_{1,x,y} - I_{2,x,y}} \right)^{2}}}}} & \left( {1\text{-}2} \right)\end{matrix}$

L1 is the pixel mean square error loss, F and H are respectively pixelvalues of a width and a height of the image, I_(1, x, y) is a pixelvalue, at a location (x, y), of the high-resolution image correspondingto the input low-resolution image, and I_(2, x, y) is a pixel value, atthe location (x, y), of the reconstructed image of the low-resolutionimage.

The image feature may be a feature extracted by an image featureextraction apparatus from the reconstructed image and thehigh-resolution image, and the image feature may be an N-dimensionalvector Θ. The feature loss may be a feature mean square error betweenthe reconstructed image and the high-resolution image, that is,

$\begin{matrix}{{L2} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}\left( {\Theta_{1,i} - \Theta_{2,i}} \right)^{2}}}} & \left( {1\text{-}3} \right)\end{matrix}$

L2 is the feature loss, Θ_(1,i) is an i^(th)-dimensional image featurevalue of the high-resolution image corresponding to the low-resolutionimage, Θ_(2,i) s an i^(th)-dimensional image feature value of thereconstructed image of the low-resolution image, and i is a positiveinteger greater than 1 and less than N.

(6) Regularization

When an initial model is constructed, a complex model is usually used tofit complex data. However, using the complex model may cause a risk ofover-fitting. Regularization is a common method in mathematicaloptimization, which can control an amplitude of a to-be-optimizedparameter, to avoid overfitting. Overfitting indicates that the modelhas a small error in a training set but a large error in a test set,that is, has a poor generalization capability. Overfitting is usuallycaused by noises in data or by using an excessively complex model to fitthe data.

In the formula (1-1), an objective is to optimize a least square error(least square error). The least square error is:

$\begin{matrix}{{E = {{\sum\limits_{i = 1}^{N}\left( {y_{i} - {f\left( x_{i} \right)}} \right)^{2}} + {\lambda W^{T}W}}},{or}} & \left( {1\text{-}4} \right) \\{E = {{\sum\limits_{i = 1}^{N}\left( {y_{i} - {f\left( x_{i} \right)}} \right)^{2}} + {\lambda {\sum{W_{j}}}}}} & \left( {1\text{-}5} \right)\end{matrix}$

f(x_(i)) is a value of the constructed model at x_(i), the constructedinitial model may be f(x_(i))=w₀x₀+w₁x₁+ . . . +w_(N)x_(N), y_(i) is asampling value, and w^(T)w. w^(T)w and Σ|W_(i)| are regularizationitems, and their objective is to reduce a risk of overfitting. W may bea weight matrix, and λ is a weight of the regularization item.

During image reconstruction, only the pixel mean square error isconsidered in the training process of the super-resolution model in theprior art. The initial super-resolution model is trained by using theerror loss including the pixel mean square error between thesuper-resolution image reconstructed from the low-resolution image inthe training set and the high-resolution image corresponding to thelow-resolution image, so that the pixel mean square error converges, toobtain a newly constructed super-resolution model. In other words, theerror loss caused by only the pixel mean square error is considered inthe newly constructed super-resolution model, and a reconstructed imageobtained by using the newly constructed super-resolution model obtainedin the foregoing training process loses much high-frequency information.This reduces reconstruction quality of the reconstructed image.

To improve the reconstruction quality of the reconstructed image, anembodiment of this application provides an image reconstruction method.In the training phase of the super-resolution model, the error lossincludes the pixel mean square error, and the error loss also includesthe image feature mean square error. The error loss used to train theinitial super-resolution model includes more comprehensive error lossinformation, so that a loss of high-frequency information of areconstructed image can be reduced, and reconstruction quality of thereconstructed image can be improved.

Inventive principles in this application may include: In the trainingphase of the super-resolution model, the low-resolution image isreconstructed by using the initial super-resolution model, to obtain areconstructed image, that is, a super-resolution image. An error lossbetween the super-resolution image and a high-resolution imagecorresponding to the low-resolution image is determined, and the errorloss includes the pixel mean square error and the image feature meansquare error. In addition, a newly constructed super-resolution model isdetermined based on the error loss and the initial super-resolutionmodel. A parameter in the initial super-resolution model is adjusted byusing a more comprehensive error loss, so that precision of imagereconstruction of the newly constructed super-resolution model can beimproved, and reconstruction quality of the reconstructed image isimproved.

Based on the foregoing main inventive principles, the followingdescribes an image reconstruction method provided in this application.

Referring to FIG. 1 and FIG. 2, FIG. 1 is a schematic flowchart of animage reconstruction method according to an embodiment of thisapplication. As shown in

FIG. 1, the image reconstruction method includes but is not limited tothe following steps S101 to S104.

S101: An image reconstruction device inputs a fifth image into aninitial super-resolution model to obtain a reconstructed third image.

S102: The image reconstruction device determines an error loss betweenthe third image and a fourth image, where the error loss includes apixel mean square error and an image feature mean square error.

S103: The image reconstruction device constructs a newly constructedsuper-resolution model based on the initial super-resolution model andthe error loss.

S104: The image reconstruction device inputs a first image into thenewly constructed super-resolution model to obtain a reconstructedsecond image.

In this embodiment of this application, an image reconstruction model isthe super-resolution model, and is used to reconstruct an image that isinput into the super-resolution model, to improve a resolution. Aprocess of establishing the super-resolution model may include atraining phase and a test phase. The training phase is a process oftraining the initial super-resolution model by using a low-resolutionimage and a high-resolution image corresponding to the low-resolutionimage, to converge an error loss. In the test phase, a low-resolutiontest image is input into the newly constructed super-resolution model toobtain a reconstructed image, and the newly constructed super-resolutionmodel may be tested to reconstruct an image, to improve an imageresolution. The test phase may also be considered as a process ofreconstructing an image by using the newly constructed super-resolutionmodel. Steps S101 to S103 may be considered as a procedure of thetraining phase of the super-resolution model. The newly constructedsuper-resolution model is a super-resolution model whose error isconverged and that is obtained by training the initial super-resolutionmodel by using the error loss and adjusting a parameter in the initialsuper-resolution model. The newly constructed super-resolution model maybe directly used to reconstruct an image to improve an image resolution.Step S104 may be considered as a process of reconstructing an image byusing the newly constructed super-resolution model, to improve the imageresolution, that is, a procedure of the test phase of the newlyconstructed super-resolution model.

Specifically, FIG. 2 is a schematic block diagram of a method forconstructing an image reconstruction model according to an embodiment ofthis application. As shown in FIG. 2, a low-resolution image 101 isinput into an initial super-resolution model 102, and the low-resolutionimage 101 is reconstructed by the initial super-resolution model 102 toobtain a reconstructed image 103. A pixel mean square error 105 betweenthe reconstructed image 103 and a high-resolution image 104 iscalculated, and an image feature mean square error 106 between thereconstructed image 103 and the high-resolution image 104 is calculated.Image features may be an image feature of the reconstructed image and animage feature of the high-resolution image that are extracted by animage feature extraction apparatus from the reconstructed image 103 andthe high-resolution image 104. An error loss 107 is determined based onthe pixel mean square error 105 and the image feature mean square error106, and a super-resolution model is updated based on the error loss 107and the initial super-resolution model 102. The low-resolution image isan image whose resolution is relatively low and that is obtained afterblurring processing is performed on the high-resolution image.

In a test phase, a first image is a low-resolution image, and a secondimage is a reconstructed image reconstructed by the reconstructedsuper-resolution model. In a training phase, a fifth image is alow-resolution image, a third image is a reconstructed imagereconstructed by the initial super-resolution model, and a fourth imageis a high-resolution image corresponding to the fifth image. The fifthimage may be a low-resolution image obtained by performing blurringprocessing on the fourth image. The fourth image and the fifth imageconstitute a training set of a high-resolution image and alow-resolution image.

In an embodiment, step S101 may be inputting M fifth images into theinitial super-resolution model to obtain M reconstructed third images.Step S102 may be determining M error losses based on the M third imagesand M fourth images, where any one of the M error losses is an errorloss between an i^(th) third image in the M third images and a j^(th)fourth image in the M fourth images, an image obtained after a fifthimage obtained by performing blurring processing on the j^(th) fourthimage is input into the initial super-resolution model is the i^(th)third image, M is an integer greater than 1, and i and j each are apositive integer less than or equal to M.

In an embodiment, in one case, the newly constructed super-resolutionmodel may be obtained by adjusting a parameter in the initialsuper-resolution model based on the M error losses.

In another case, the initial super-resolution model is the firstsuper-resolution model, a parameter in the first super-resolution modelis adjusted based on the first error loss in the M error losses toobtain the second super-resolution model, a parameter in an r^(th)super-resolution model is adjusted based on an r^(th) error loss toobtain an (r+1)^(th) super-resolution model, and the newly constructedsuper-resolution model is obtained by adjusting a parameter in an M^(th)super-resolution model by using an M^(th) error loss, where r is apositive integer greater than or equal to 1 and less than or equal to M.

In other words, there may be a plurality of pairs of training samples,and the M error losses may be obtained through calculation by using Mpairs of training samples. In the first case, the initialsuper-resolution model is adjusted once by using the M error losses, toobtain the newly constructed super-resolution model. In the second case,each time an error loss in the M error losses is obtained, thesuper-resolution model is adjusted by using the error loss, that is, thesuper-resolution model is adjusted for M times to obtain the newlyconstructed super-resolution model. When M is a positive integer greaterthan 2, the initial super-resolution model is adjusted once by using aplurality of error losses obtained by using a plurality of groups oftraining samples, to obtain the newly constructed super-resolutionmodel, so that more sample information is provided for adjusting theparameter in the initial super-resolution model, and the newlyconstructed super-resolution model obtained through adjustment hashigher precision. In addition, if there are a plurality of pairs oftraining samples, and each time an error loss is obtained, the initialsuper-resolution model is adjusted based on the error loss, an excessivequantity of adjustment times causes wastes of processing resources andstorage resources. However, if the parameter in the initialsuper-resolution model is adjusted once by using the plurality of errorlosses obtained by using the plurality of groups of training samples, toobtain the newly constructed super-resolution model, a quantity of timesof adjusting the parameter in the super-resolution model can be reduced.Therefore, processing resources and storage resources can be saved.

It should be noted that a method for training the initialsuper-resolution model based on the plurality of error losses to obtainthe newly constructed super-resolution model is not limited to theforegoing two methods, and another method may be used to train theinitial super-resolution model based on the plurality of error losses toobtain the newly constructed super-resolution model. For example, thesuper-resolution model is trained for N times, and at least one traininguses more than one error loss. N may be a positive integer greater than1 and less than M. The methods provided in the foregoing two cases aremerely used to explain this embodiment of this application, and shouldnot constitute a limitation.

In an embodiment, the super-resolution model may include a plurality ofsuper-resolution submodels. FIG. 3 is a schematic structural diagram ofa super-resolution model according to an embodiment of this application.As shown in FIG. 3, the super-resolution model 102 may include nsuper-resolution submodels 1021, and n is a positive integer greaterthan or equal to 2. The super-resolution submodel 1021 is used toreconstruct image information input into the super-resolution submodel1021, to improve a resolution. The image information includes pixelvalue information and image feature information.

In the n super-resolution submodels, an input of the firstsuper-resolution submodel 1 is a first image, that is, a low-resolutionimage 101, an output of the first super-resolution submodel 1 is used asan input of the second super-resolution submodel 2, an output of a(t−1)^(th) super-resolution submodel t−1 is used as an input of a t^(th)super-resolution submodel t, and an output of the t^(th)super-resolution submodel t is used as an input of a (t+1)^(th)super-resolution submodel t+1; t is a positive integer satisfying2≤t≤n−1; and the output of the t^(th) super-resolution submodel t isused as an input of an output synthesis module 1022, an output of theoutput synthesis module 1022 is used as an input of an n^(th)super-resolution submodel n, an output of the n^(th) super-resolutionsubmodel n is a second image, that is, a reconstructed image 103, andthe output synthesis module 1022 is configured to determine imageinformation input into the n^(th) super-resolution submodel n based onreconstructed image information output by the first n−1 super-resolutionsubmodels 1021 and a weight of each piece of the output reconstructedimage information.

The super-resolution model 102 may be included in an imagereconstruction device, so that the image reconstruction device performsthe image reconstruction method described in FIG. 1.

When n is a positive integer greater than 1, a plurality ofsuper-resolution submodels are cascaded to reconstruct a low-resolutionimage. A pixel value of a reconstructed image obtained throughreconstruction is higher, so that image quality of the reconstructedimage can be improved. In addition, the reconstructed images output bythe first n−1 super-resolution submodels are all used as imageinformation input into the last super-resolution submodel, and includemore image information, so that image information loss is reduced. Thiscan improve image reconstruction quality by improving precision of thenewly constructed super-resolution model.

When the super-resolution model is actually designed, a qualityrequirement on the reconstructed image and calculation resources need tobe considered for a quantity of the super-resolution submodels. A higherquality requirement on the reconstructed image requires a largerquantity of the super-resolution submodels. However, the larger quantityof the super-resolution submodels causes a larger system calculationamount, and consumes more calculation resources. Therefore, the quantityof the super-resolution submodels needs to be selected based on atradeoff between the quality requirement on the reconstructed image andthe calculation resources.

For example, as shown in FIG. 3, it is assumed that image informationthat can be reconstructed by the first super-resolution submodel isabout 90% of image information of the high-resolution image in thetraining set, the 90% image information is first image information,image information that can be reconstructed by the secondsuper-resolution submodel is about 90% of image information other thanthe first image information, and so on. In other words, in an imagereconstruction process of each super-resolution submodel, 10% imageinformation of remaining image information that is not reconstructed islost. The remaining image information that is not reconstructed may beunderstood as image information lost by a previous super-resolutionsubmodel. For the first super-resolution submodel, the remaining imageinformation that is not reconstructed may be understood as all imageinformation of the high-resolution image. If the quality requirement onthe reconstructed image is to reconstruct image information with imagequality of 99% of the high-resolution image, and the calculationresources can process a maximum of five super-resolution submodels inreal time, assuming that the quantity of the super-resolution submodelsis and N is a positive integer greater than or equal to 1, N satisfiesthe following condition:

$\begin{matrix}\left\{ \begin{matrix}{{1 - \left( {1 - 0.9} \right)^{N}} > 0.99} \\{1 \leq N \leq 5}\end{matrix} \right. & \left( {1\text{-}6} \right)\end{matrix}$

According to the foregoing formula (1-6), N=2 may be solved. In thiscase, the quantity of the super-resolution submodels may be set to 2.

It may be understood that, if there is one or two super-resolutionsubmodels, the super-resolution model may not include the outputsynthesis module 1022, or may include the output synthesis module 1022.When there are two super-resolution submodels, the two super-resolutionsubmodels may be cascaded. An input of the first super-resolutionsubmodel is a low-resolution image, and a reconstructed imageinformation output by the first super-resolution submodel is used as aninput of the second super-resolution submodel, an output of the secondsuper-resolution submodel is a reconstructed image of thesuper-resolution model.

As shown in FIG. 3, reconstructed image information output by the outputsynthesis module 1022 is:

$\begin{matrix}{O_{S} = {\sum\limits_{k = 1}^{n - 1}\left( {w_{k}O_{k}} \right)}} & \left( {1\text{-}7} \right)\end{matrix}$

k is a positive integer satisfying 1≤k≤n−1; and w_(k) is a weight of ak_(th) super-resolution submodel.

In the super-resolution model, the weight w_(k) of the k_(th)super-resolution submodel may be determined based on an experimentalresult or experience. The weight w_(k) of the k_(th) super-resolutionsubmodel may alternatively be a parameter in the super-resolution model.In other words, in the training process of the initial super-resolutionmodel, the weight w_(k) in the initial super-resolution model may beoptimized based on the error loss.

In an embodiment, the super-resolution submodel 1021 may be athree-layer fully convolutional deep neural network. In the three-layerconvolutional deep neural network, the first convolution layer may be aninput layer, and is used to extract image information by region. Theinput layer may include a plurality of convolution kernels, used toextract different image information. The second convolution layer may bea transform layer, and is used to perform non-linear transform on theextracted image information. If the extracted image information is X,the non-linear transform may be f(W·X+b). The image information X may bea multidimensional vector, and is multidimensional image informationextracted by using the plurality of convolution kernels at the firstconvolution layer. W is a weight vector of the convolution kernel, b isa bias vector of the convolution kernel, and f may be an activationfunction. The third convolution layer may be an output layer, and isused to reconstruct image information output by the second convolutionlayer. Alternatively, the reconstruction process may be performingreconstruction by performing a convolution operation on an image byusing the plurality of convolution kernels. An output of the outputlayer may be a 3-channel (color) image or a single-channel (grayscale)image.

If the super-resolution model includes a plurality of super-resolutionsubmodels that are cascaded, in the foregoing three-layer fullyconvolutional deep neural network, the first convolution layer and thesecond convolution layer are used to extract image information from alow-resolution image, that is, obtain information that can be used forsuper-resolution reconstruction. The third convolution layerreconstructs a high-resolution image by using the image informationextracted and transformed by the first two layers. The two additionalconvolution layers in the three-layer fully convolutional deep neuralnetwork can extract more precise image information than extracting theimage information by using only one convolution layer. In addition, thesuper-resolution submodels constituted by three-layer fullyconvolutional deep neural networks need to be cascaded to constitute thesuper-resolution model, and cascading a plurality of super-resolutionsubmodels requires more calculation resources, but a relatively smallquantity of convolution layers indicates a relatively low calculationamount. Therefore, a tradeoff between calculation resources andprecision needs to be considered for a quantity of convolution layers inthe super-resolution submodels. When the super-resolution submodel usesthe three-layer fully convolutional deep neural network, more preciseimage information can be extracted by using fewer calculation resources.The more precise image information helps reconstruct a higher-qualityreconstructed image and save calculation resources.

In an embodiment, the weight vector W of the convolution kernel may be aparameter in the super-resolution model. In other words, in the trainingprocess of the initial super-resolution model, the weight vector W ofthe convolution kernel may be optimized based on the error loss.

In an embodiment, the error loss is:

L=λ ₁ L1+λ₂ L2+λ₃ L3   (1-8)

L1 is a pixel mean square error. For details, refer to the formula(1-2). λ₁ is a weight of the pixel mean square error. L2 is an imagefeature mean square error. For details, refer to the formula (1-3). λ₂is a weight of the image feature mean square error, L3 is aregularization term of w_(k), and λ₃ is a weight of the regularizationterm.

w=(w ₁ , w ₂ , w ₃ , . . . , w _(N−1))   (1-9)

w is a weight matrix of the super-resolution submodel.

L3=w ^(T) w, or L3=Σ|w _(i)|  (1-10)

Values of λ₁, λ₂, and λ₃ may be determined based on an experiment orexperience.

The added regularization term L3 is used to reduce overfitting, improveprecision of the newly constructed super-resolution model, and improvequality of the reconstructed image.

For example, in a face image reconstruction scenario, first, preparationbefore training needs to be performed, that is, a training set needs tobe obtained. A specific process is as follows:

Step 1: An obtained high-resolution face image is {Y^(m)|1≤m≤M}εR^(a×b).

M is a quantity of training samples, R^(a×b) indicates a size of theimage, and a resolution of the high-resolution face image is a×b.

Step 2: Obtain a low-resolution face image T^(m)=D(Y^(m)) by using adown-sampling function, where {T^(m)|1≤m≤M}εR^((a/t)×(b/t)).

D may be a down-sampling function, that is, a fuzzy function. Aresolution of the low-resolution face image is (a/t)×(b/t), t is apositive integer, {T^(m)|1≤m≤M}εR^((a/t)×(b/t)), and{Y^(m)|1≤m≤M}εR^(a×b), that is, {T^(m)|1≤m≤M}εR^((a/t)×(b/t))and{Y^(m)|1≤m≤M}εR^(a×b) constitute the training set.

Then, the super-resolution model is trained based on the training set. Aspecific process is as follows:

As shown in FIG. 3, the super-resolution model may include nsuper-resolution submodels. FIG. 4 is a schematic structural diagram ofa super-resolution model according to an embodiment of this application.As shown in FIG. 4, a k^(th) super-resolution submodel may be athree-layer fully convolutional deep neural network. An input of thek^(th) super-resolution submodel is a face image 104, and the face image104 may be a face image output by a (k−1)^(th) super-resolutionsubmodel. If k=1, the face image 104 may be the low-resolution image 101input into the super-resolution model 102 in FIG. 3. To be specific, theface image output by the (k−1)^(th) super-resolution submodel or thelow-resolution face image is used as an input of the first convolutionlayer in the k^(th) super-resolution submodel, an output of the firstconvolution layer in the k^(th) super-resolution submodel is face imageinformation X_(k) obtained by performing a convolution operation byusing each of s convolution kernels, and the face image informationX_(k) is used as an input of the second convolution layer in the k^(th)super-resolution submodel. Xk=(X1, X2, . . . , Xs), an output of thesecond convolution layer in the k^(th) super-resolution submodel isf(W·X+b), and an output of the third convolution layer in the k^(th)super-resolution submodel is a reconstructed face image 105 obtained byperforming convolution calculation by using f(W·X+b) and m convolutionkernels.

It should be noted that a size of each convolution kernel at the firstconvolution layer may be different from a size of a convolution kernelat the third convolution layer, and a size of a convolution kernel atthe second convolution layer may be 1. A quantity of convolution kernelsat the first convolution layer, a quantity of convolution kernels at thesecond convolution layer, and a quantity of convolution kernels at thethird convolution layer may be the same or different. k is a positiveinteger satisfying 1≤k≤n−1.

The n^(th) super-resolution submodel may also be a three-layer fullyconvolutional deep neural network. Face image information input into thefirst convolution layer of the n^(th) super-resolution submodel may be

${O_{S} = {\sum\limits_{k = 1}^{n - 1}\left( {w_{k}O_{k}} \right)}},$

where O_(k) is face image information of the reconstructed face image ofthe k^(th) super-resolution submodel. The second convolution layer ofthe n^(th) super-resolution submodel is similar to the secondconvolution layer of the k^(th) super-resolution submodel. Details arenot described again. A reconstructed face image 105 output by the thirdconvolution layer of the n^(th) super-resolution submodel is thereconstructed image 103 in the super-resolution model described in FIG.3.

In the n^(th) super-resolution submodel, a quantity of convolutionkernels at the third convolution layer may be the same as a quantity ofchannels of the input low-resolution face image, and the convolutionkernels are used to reconstruct the low-resolution face image, to obtainthe reconstructed face image. For example, if the low-resolution faceimage has three channels, that is, R, G, and B each occupy one channel,there are three convolution kernels at the third convolution layer ofthe last super-resolution submodel, that is, the n^(th) super-resolutionsubmodel, and the three convolution kernels are used to reconstruct thelow-resolution face image including three colors: red, green, and blue,to obtain the reconstructed face image. The reconstructed face imagealso consists of red, green, and blue. For another example, if thelow-resolution face image has one channel, that is, the low-resolutionface image is a grayscale image, there is one convolution kernel at thethird convolution layer of the last super-resolution submodel, that is,the n^(th) super-resolution submodel, and the convolution kernel is usedto reconstruct a grayscale low-resolution face image, to obtain areconstructed face image. The reconstructed face image is also agrayscale image.

In an embodiment, after the error loss is obtained, the parameters inthe newly constructed super-resolution model may alternatively bedetermined based on the error loss and at least one of the following:the first image, the second image, and the third image. In other words,a device for constructing the image reconstruction model may determinethe parameter in the newly constructed super-resolution model based onthe error loss and an image related to the error loss, and does not needto adjust a parameter value based on the parameter in the initialsuper-resolution model.

It should be noted that the foregoing image reconstruction method may beapplied to an image recognition system, for example, a facialrecognition system. The foregoing image reconstruction method may beapplied to an image intensifier system.

The methods in the embodiments of the present invention are describedabove in detail, and apparatuses in the embodiments of the presentinvention are provided below.

FIG. 5 is a schematic structural diagram of an image reconstructiondevice according to an embodiment of this application. As shown in FIG.5, the device may include a processing module 501 and a receive module502.

The processing module 501 inputs a first image into a newly constructedsuper-resolution model to obtain a reconstructed second image, where aresolution of the second image is higher than that of the first image;

the newly constructed super-resolution model is obtained by training aninitial super-resolution model by using an error loss; the error lossincludes a pixel mean square error and an image feature mean squareerror; and an image feature includes at least one of a texture feature,a shape feature, a spatial relationship feature, and an image high-levelsemantic feature; and

the receive unit 502 is configured to receive the first image that isinput into the newly constructed super-resolution model. In a possibleimplementation, the error loss is an error loss between a third imageand a fourth image, and the third image is obtained throughreconstruction after inputting a fifth image into the initialsuper-resolution model; the fourth image is a high-resolution image, andthe fifth image is a low-resolution image obtained by performingblurring processing on the fourth image;

and the initial super-resolution model is used to reconstruct an imageinput into the initial super-resolution model, to improve a resolution.

In a possible implementation, there are M third images, M fourth images,and M fifth images, there are M error losses, and the M third images areobtained through reconstruction after inputting the M fifth images intothe initial super-resolution model; the M error losses are determinedbased on the M third images and the M fourth images; and

any one of the M error losses is an error loss between an i^(th) thirdimage in the M third images and a j^(th) fourth image in the M fourthimages, an image obtained after a fifth image obtained by performingblurring processing on the j^(th) fourth image is input into the initialsuper-resolution model is the i^(th) third image, M is a positiveinteger greater than 1, and i and j each are a positive integer lessthan or equal to M.

In a possible implementation, the newly constructed super-resolutionmodel is obtained by adjusting a parameter in the initialsuper-resolution model based on the M error losses; or

the initial super-resolution model is the first super-resolution model,a parameter in the first super-resolution model is adjusted based on thefirst error loss in the M error losses to obtain the secondsuper-resolution model, a parameter in an r^(th) super-resolution modelis adjusted based on an r^(th) error loss to obtain an (r+1)^(th)super-resolution model, and the newly constructed super-resolution modelis obtained by adjusting a parameter in an M^(th) super-resolution modelby using an M^(th) error loss, where r is a positive integer greaterthan or equal to 1 and less than or equal to M.

In a possible implementation, the initial super-resolution modelincludes n super-resolution submodels, and n is a positive integergreater than or equal to 2; the super-resolution submodel is used toreconstruct image information input into the super-resolution submodel,to improve a resolution; the image information includes pixel valueinformation and image feature information;

in the n super-resolution submodels, an input of the firstsuper-resolution submodel is the first image, an output of the firstsuper-resolution submodel is used as an input of the secondsuper-resolution submodel, an output of a (t−1)^(th) super-resolutionsubmodel is used as an input of a t^(th) super-resolution submodel, andan output of the t^(th) super-resolution submodel is used as an input ofa (t+1)^(th) super-resolution submodel; t is a positive integersatisfying 2≤t≤n−1; and the output of the t^(th) super-resolutionsubmodel is used as an input of an output synthesis module, an output ofthe output synthesis module is used as an input of an n^(th)super-resolution submodel, an output of the n^(th) super-resolutionsubmodel is the second image, and the output synthesis module isconfigured to determine the input of the n^(th) super-resolutionsubmodel based on reconstructed image information output by the firstn−1 super-resolution submodels and a weight of each piece of the outputreconstructed image information. In other words, the foregoing initialsuper-resolution model is included in the processing unit 501.

In a possible implementation, the reconstructed image information outputby the output synthesis module

${O_{S} = {\sum\limits_{k = 1}^{n - 1}\left( {w_{k}O_{k}} \right)}},$

and k is a positive integer satisfying 1≤k≤n−1; and w_(k) is a weight ofa k^(th) super-resolution submodel.

In a possible implementation, w_(k) is the parameter in thesuper-resolution model.

In a possible implementation, the super-resolution submodel is athree-layer fully convolutional deep neural network.

In a possible implementation, the error loss L=λ₁L1+λ₂L2+λ₃L3, where L1is the pixel mean square error, λ₁ is a weight of the pixel mean squareerror, L2 is the image feature mean square error, λ₂ is a weight of theimage feature mean square error, L3 is a regularization term of w_(k),and λ₃ is a weight of the regularization term.

It should be noted that for implementation of each module, refer to thecorresponding descriptions of the method embodiment shown in FIG. 1.Details are not described herein again.

The foregoing image reconstruction device may be an image recognitiondevice, for example, a facial recognition device. The imagereconstruction device may also be an image intensifier device or thelike.

FIG. 6 is a schematic structural diagram of another image reconstructiondevice according to an embodiment of this application. As shown in FIG.6, the device includes a processor 601, a memory 602, and acommunications interface 603. The processor 601, the memory 602, and thecommunications interface 603 are connected to each other by using a bus604.

The memory 602 includes but is not limited to a random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM), or a portable read-only memory (CD-ROM). The memory 602is configured to store a related instruction and related data.Specifically, the memory 602 may be configured to store asuper-resolution model.

The communications interface 603 may be configured to communicate withanother device, for example, may be configured to receive a trainingset, where the training set includes a fifth image and a fourth image,the fourth image is a high-resolution image, and the fifth image is alow-resolution image obtained after blurring processing is performed onthe fourth image. The communications interface 603 may be furtherconfigured to receive a low-resolution image that needs to bereconstructed, for example, receive a first image.

The processor 601 may be one or more central processing units (CPU).When the processor 601 is one CPU, the CPU may be a single-core CPU or amulti-core CPU.

The processor 601 in the image reconstruction device is configured toread program code stored in the memory 602, to perform the followingoperations:

inputting a first image into a newly constructed super-resolution modelto obtain a reconstructed second image, where a resolution of the secondimage is higher than that of the first image;

the newly constructed super-resolution model is obtained by training aninitial super-resolution model by using an error loss; the error lossincludes a pixel mean square error and an image feature mean squareerror; and an image feature includes at least one of a texture feature,a shape feature, a spatial relationship feature, and an image high-levelsemantic feature.

In a possible implementation, the error loss is an error loss between athird image and a fourth image, and the third image is obtained throughreconstruction after inputting a fifth image into the initialsuper-resolution model; the fourth image is a high-resolution image, andthe fifth image is a low-resolution image obtained by performingblurring processing on the fourth image; and the initialsuper-resolution model is used to reconstruct an image input into theinitial super-resolution model, to improve a resolution.

In a possible implementation, there are M third images, M fourth images,and M fifth images, there are M error losses, and the M third images areobtained through reconstruction after inputting the M fifth images intothe initial super-resolution model; the M error losses are determinedbased on the M third images and the M fourth images; and

any one of the M error losses is an error loss between an i^(th) thirdimage in the M third images and a j^(th) fourth image in the M fourthimages, an image obtained after a fifth image obtained by performingblurring processing on the j^(th) fourth image is input into the initialsuper-resolution model is the i^(th) third image, M is a positiveinteger greater than 1, and i and j each are a positive integer lessthan or equal to M.

In a possible implementation, the newly constructed super-resolutionmodel is obtained by adjusting a parameter in the initialsuper-resolution model based on the M error losses; or

the initial super-resolution model is the first super-resolution model,a parameter in the first super-resolution model is adjusted based on thefirst error loss in the M error losses to obtain the secondsuper-resolution model, a parameter in an r^(th) super-resolution modelis adjusted based on an r^(th) error loss to obtain an (r+1)^(th)super-resolution model, and the newly constructed super-resolution modelis obtained by adjusting a parameter in an M^(th) super-resolution modelby using an M^(th) error loss, where r is a positive integer greaterthan or equal to 1 and less than or equal to M.

In a possible implementation, the initial super-resolution modelincludes n super-resolution submodels, and n is a positive integergreater than or equal to 2; the super-resolution submodel is used toreconstruct image information input into the super-resolution submodel,to improve a resolution; the image information includes pixel valueinformation and image feature information;

in the n super-resolution submodels, an input of the firstsuper-resolution submodel is the first image, an output of the firstsuper-resolution submodel is used as an input of the secondsuper-resolution submodel, an output of a (t−1)^(th) super-resolutionsubmodel is used as an input of a t^(th) super-resolution submodel, andan output of the t^(th) super-resolution submodel is used as an input ofa (t+1)^(th) super-resolution submodel; t is a positive integersatisfying 2≤t≤n−1; and the output of the t^(th) super-resolutionsubmodel is used as an input of an output synthesis module, an output ofthe output synthesis module is used as an input of an n^(th)super-resolution submodel, an output of the n^(th) super-resolutionsubmodel is the second image, and the output synthesis module isconfigured to determine the input of the n^(th) super-resolutionsubmodel based on reconstructed image information output by the firstn−1 super-resolution submodels and a weight of each piece of the outputreconstructed image information.

In a possible implementation, the reconstructed image information outputby the output synthesis module

${O_{S} = {\sum\limits_{k = 1}^{n - 1}\left( {w_{k}O_{k}} \right)}},$

and k is a positive integer satisfying 1≤k≤n−1; and w_(k) is a weight ofa k^(th) super-resolution submodel.

In a possible implementation, w_(k) is the parameter in thesuper-resolution model.

In a possible implementation, the super-resolution submodel is athree-layer fully convolutional deep neural network.

In a possible implementation, the error loss L=λ₁L1+λ₂L2+λ₃L3, where L1is the pixel mean square error, λ₁ is a weight of the pixel mean squareerror, L2 is the image feature mean square error, λ₂ is a weight of theimage feature mean square error, L3 is a regularization term of w_(k),and λ₃ is a weight of the regularization term.

It should be noted that for implementation of each foregoing operation,refer to the corresponding descriptions of the method embodiment shownin FIG. 1. Details are not described herein again. The foregoing imagereconstruction device may be an image recognition device, for example, afacial recognition device. The image reconstruction device may also bean image intensifier device or the like.

An embodiment of the present invention further provides a chip system.The chip system includes at least one processor, a memory, and aninterface circuit. The memory, the interface circuit, and the at leastone processor are interconnected by using a line, and the at least onememory stores an instruction. When the instruction is executed by theprocessor, the method procedure shown in FIG. 1 is implemented.

An embodiment of the present invention further provides acomputer-readable storage medium. The computer-readable storage mediumstores an instruction, and when the instruction is run on a processor,the method procedure shown in FIG. 1 is implemented.

An embodiment of the present invention further provides a computerprogram product. When the computer program product runs on a processor,the method procedure shown in FIG. 1 is implemented.

All or some of the foregoing embodiments may be implemented by usingsoftware, hardware, firmware, or any combination thereof. When softwareis used to implement the embodiments, the embodiments may be implementedcompletely or partially in a form of a computer program product. Thecomputer program product includes one or more computer instructions.When the computer program instructions are loaded and executed on thecomputer, the procedure or functions according to the embodiments ofthis application are all or partially generated. The computer may be ageneral-purpose computer, a dedicated computer, a computer network, orother programmable apparatuses. The computer instructions may be storedin a computer-readable storage medium or may be transmitted from acomputer-readable storage medium to another computer-readable storagemedium. For example, the computer instructions may be transmitted from awebsite, computer, server, or data center to another website, computer,server, or data center in a wired (for example, a coaxial cable, anoptical fiber, or a digital subscriber line (DSL)) or wireless (forexample, infrared, radio, or microwave) manner. The computer-readablestorage medium may be any usable medium accessible by a computer, or adata storage device, such as a server or a data center, integrating oneor more usable media. The usable medium may be a magnetic medium (forexample, a floppy disk, a hard disk, or a magnetic tape), an opticalmedium (for example, a DVD), a semiconductor medium (for example, asolid-state drive (SSD)), or the like.

A person of ordinary skill in the art may understand that all or some ofthe processes of the methods in the embodiments may be implemented by acomputer program instructing relevant hardware. The program may bestored in a computer readable storage medium. When the program runs, theprocesses of the methods in the embodiments are performed. The foregoingstorage medium includes: any medium that can store program code, such asa ROM or a random access memory RAM, a magnetic disk or an optical disc.

What is claimed is:
 1. An image reconstruction method, comprising:inputting a first image into a newly constructed super-resolution modelto obtain a reconstructed second image, wherein a resolution of thesecond image is higher than that of the first image; the newlyconstructed super-resolution model is obtained by training an initialsuper-resolution model by using an error loss; the error loss comprisesa pixel mean square error and an image feature mean square error; and animage feature comprises at least one of a texture feature, a shapefeature, a spatial relationship feature, or an image high-level semanticfeature.
 2. The method according to claim 1, wherein the error loss isan error loss between a third image and a fourth image, and the thirdimage is obtained through reconstruction after inputting a fifth imageinto the initial super-resolution model; the fourth image is ahigh-resolution image, and the fifth image is a low-resolution imageobtained by performing blurring processing on the fourth image; and theinitial super-resolution model is used to reconstruct an image inputinto the initial super-resolution model, to improve a resolution.
 3. Themethod according to claim 2, wherein there are M third images, M fourthimages, and M fifth images, there are M error losses, and the M thirdimages are obtained through reconstruction after inputting the M fifthimages into the initial super-resolution model; the M error losses aredetermined based on the M third images and the M fourth images; and anyone of the M error losses is an error loss between an i^(th) third imagein the M third images and a j^(th) fourth image in the M fourth images,an image obtained after a fifth image obtained by performing blurringprocessing on the j^(th) fourth image is input into the initialsuper-resolution model is the i^(th) third image, M is a positiveinteger greater than 1, and i and j each are a positive integer lessthan or equal to M.
 4. The method according to claim 3, wherein thenewly constructed super-resolution model is obtained by adjusting aparameter in the initial super-resolution model based on the M errorlosses; or the initial super-resolution model is the firstsuper-resolution model, a parameter in the first super-resolution modelis adjusted based on the first error loss in the M error losses toobtain the second super-resolution model, a parameter in an r^(th)super-resolution model is adjusted based on an r^(th) error loss toobtain an (r+1)^(th) super-resolution model, and the newly constructedsuper-resolution model is obtained by adjusting a parameter in an M^(th)super-resolution model by using an M^(th) error loss, wherein r is apositive integer greater than or equal to 1 and less than or equal to M.5. The method according to claim 1, wherein the initial super-resolutionmodel comprises n super-resolution submodels, and n is a positiveinteger greater than or equal to 2; the super-resolution submodel isused to reconstruct image information input into the super-resolutionsubmodel, to improve a resolution; the image information comprises pixelvalue information and image feature information; in the nsuper-resolution submodels, an input of the first super-resolutionsubmodel is the first image, an output of the first super-resolutionsubmodel is used as an input of the second super-resolution submodel, anoutput of a (t−1)^(th) super-resolution submodel is used as an input ofa t^(th) super-resolution submodel, and an output of the t^(th)super-resolution submodel is used as an input of a (t+1)^(th)super-resolution submodel; t is a positive integer satisfying 2≤t≤n−1;and the output of the t^(th) super-resolution submodel is used as aninput of an output synthesis module, an output of the output synthesismodule is used as an input of an n^(th) super-resolution submodel, anoutput of the n^(th) super-resolution submodel is the second image, andthe output synthesis module is configured to determine the input of then^(th) super-resolution submodel based on reconstructed imageinformation output by the first n−1 super-resolution submodels and aweight of each piece of the output reconstructed image information. 6.The method according to claim 5, wherein the reconstructed imageinformation output by the output synthesis module${O_{S} = {\sum\limits_{k = 1}^{n - 1}\left( {w_{k}O_{k}} \right)}},$and k is a positive integer satisfying 1≤k≤n−1; and w_(k) is a weight ofa k^(th) super-resolution submodel.
 7. The method according to claim 6,wherein w_(k) is a parameter in the initial super-resolution model. 8.The method according to claim 5, wherein the super-resolution submodelis a three-layer fully convolutional deep neural network.
 9. The methodaccording to claim 6, wherein the error loss L=λ₁L1+λ₂L2+λ₃L3, whereinL1 is the pixel mean square error, λ₁ is a weight of the pixel meansquare error, L2 is the image feature mean square error, λ₂ is a weightof the image feature mean square error, L3 is a regularization term ofw_(k), and λ₃ is a weight of the regularization term.
 10. An imagereconstruction device, comprising a processor and a memory, wherein thememory is configured to store a program instruction, and the processoris configured to invoke the program instruction to perform the followingoperations: inputting a first image into a newly constructedsuper-resolution model to obtain a reconstructed second image, wherein aresolution of the second image is higher than that of the first image;the newly constructed super-resolution model is obtained by training aninitial super-resolution model by using an error loss; the error losscomprises a pixel mean square error and an image feature mean squareerror; and an image feature comprises at least one of a texture feature,a shape feature, a spatial relationship feature, or an image high-levelsemantic feature.
 11. The device according to claim 10, wherein theerror loss is an error loss between a third image and a fourth image,and the third image is obtained through reconstruction after inputting afifth image into the initial super-resolution model; the fourth image isa high-resolution image, and the fifth image is a low-resolution imageobtained by performing blurring processing on the fourth image; and theinitial super-resolution model is used to reconstruct an image inputinto the initial super-resolution model, to improve a resolution. 12.The device according to claim 11, wherein there are M third images, Mfourth images, and M fifth images, there are M error losses, and the Mthird images are obtained through reconstruction after inputting the Mfifth images into the initial super-resolution model; the M error lossesare determined based on the M third images and the M fourth images; andany one of the M error losses is an error loss between an i^(th) thirdimage in the M third images and a j^(th) fourth image in the M fourthimages, an image obtained after a fifth image obtained by performingblurring processing on the j^(th) fourth image is input into the initialsuper-resolution model is the i^(th) third image, M is a positiveinteger greater than 1, and i and j each are a positive integer lessthan or equal to M.
 13. The device according to claim 12, wherein thenewly constructed super-resolution model is obtained by adjusting aparameter in the initial super-resolution model based on the M errorlosses; or the initial super-resolution model is the firstsuper-resolution model, a parameter in the first super-resolution modelis adjusted based on the first error loss in the M error losses toobtain the second super-resolution model, a parameter in an r^(th)super-resolution model is adjusted based on an r^(th) error loss toobtain an (r+1)^(th) super-resolution model, and the newly constructedsuper-resolution model is obtained by adjusting a parameter in an M^(th)super-resolution model by using an M^(th) error loss, wherein r is apositive integer greater than or equal to 1 and less than or equal to M.14. The device according to claim 10, wherein the initialsuper-resolution model comprises n super-resolution submodels, and n isa positive integer greater than or equal to 2; the super-resolutionsubmodel is used to reconstruct image information input into thesuper-resolution submodel, to improve a resolution; the imageinformation comprises pixel value information and image featureinformation; in the n super-resolution submodels, an input of the firstsuper-resolution submodel is the first image, an output of the firstsuper-resolution submodel is used as an input of the secondsuper-resolution submodel, an output of a (t−1)^(th) super-resolutionsubmodel is used as an input of a t^(th) super-resolution submodel, andan output of the t^(th) super-resolution submodel is used as an input ofa (t+1)^(th) super-resolution submodel; t is a positive integersatisfying 2≤t≤n−1; and the output of the t^(th) super-resolutionsubmodel is used as an input of an output synthesis module, an output ofthe output synthesis module is used as an input of an n^(th)super-resolution submodel, an output of the n^(th) super-resolutionsubmodel is the second image, and the output synthesis module isconfigured to determine the input of the n^(th) super-resolutionsubmodel based on reconstructed image information output by the firstn−1 super-resolution submodels and a weight of each piece of the outputreconstructed image information.
 15. The device according to claim 14,wherein the reconstructed image information output by the outputsynthesis module${O_{S} = {\sum\limits_{k = 1}^{n - 1}\left( {w_{k}O_{k}} \right)}},$and k is a positive integer satisfying 1≤k≤n−1; and w_(k) is a weight ofa k^(th) super-resolution submodel.
 16. The device according to claim15, wherein w_(k) is a parameter in the initial super-resolution model.17. The device according to claim 14, wherein the super-resolutionsubmodel is a three-layer fully convolutional deep neural network. 18.The device according to claim 15, wherein the error lossL=λ₁L1+λ₂L2+λ₃L3, wherein L1 is the pixel mean square error, λ₁ is aweight of the pixel mean square error, L2 is the image feature meansquare error, λ₂ is a weight of the image feature mean square error, L3is a regularization term of w_(k), and λ₃ is a weight of theregularization term.
 19. A computer-readable storage medium, wherein thecomputer-readable storage medium stores a program instruction, and whenthe program instruction is run by a processor, the method according toclaim 1 is implemented.